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ABSTRACT 


Personalizing learning with technology in secondary schools might be a way to diffuse innovations in both technology 
and education at the same time. In the current study, personalizing learning with technology is studied from three 
perspectives: teacher, learner and technology. Data about the implementation and evaluation of the interventions in 
school were gathered by interviews with teachers and students, teacher logbooks and teacher and student questionnaires. 
Moreover, test and questionnaire data were collected on achievement, school motivation and self-regulation of more than 
4800 students. Effects of 35 personalizing learning interventions with mobile technology in 27 secondary schools in the 
Netherlands were examined. Generally, three types of personalizing learning interventions seem to increase student 
achievement: 1) a comprehensive approach across the school organization and programs; 2) personalizing learning with 
teachers differentiating either convergently or divergently, and 3) learner-control interventions in which students have 
control of surface aspects such as pacing, sequencing and practicing within limits set by teachers or programs. Too much 
emphasis on learner control instead of teacher control does not seem to benefit cognitive outcomes. The conclusions with 
respect to students’ motivation and self-regulation are less clear-cut. 
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1. INTRODUCTION 


The increasing use of technology in society in combination with attention for constructivist learning 
orientations requires secondary schools to change to more suitable learning and teaching practices. Yet 
innovations in teaching with technology have entered the school sporadically: most classroom teachers use 
the technology to do what they always have done and choose those activities that will help them 
accommodate their own perspectives on teaching and learning (Liu, 2011). Many schools have integrated 
laptops and other digital tools into daily practice, but it is unclear if those devices are being used in ways that 
best maximize their potential (Greaves, et al. 2010). Despite increased prevalence and use of computers in 
schools, research on the effectiveness has yielded mixed results (Machin et al., 2007). This problem 
motivated the Dutch government to setup a tender to stimulate secondary schools to diffuse innovations in 
both technology and education at the same time. Schools were invited to submit their proposals to receive 
funds to develop and implement various school-based innovations on personalizing learning with technology. 
The objective of this study is to provide insights into the effects of the various approaches to personalizing 
learning with technology in secondary schools. 


2. PERSONALIZING LEARNING WITH TECHNOLOGY 


In contrast to traditional classroom instruction based on teacher-centered approaches, the use of technology 
allows educators to empower students to take control of their own learning. Learner control is the degree to 
which students can direct their own learning experiences (Shyu and Brown, 1992), including path, pace, and 
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instructional approach. Such control could include choices at the curriculum level (sequence of instructional 
materials), the opportunity to choose how long to focus on a learning objective (pacing), or the ability to 
select and sequence a variety of review strategies (choice of practice items or amount of review material; 
Niemiec et al., 1996). As a result of making their own instructional decisions while learning, students would 
be more likely to explore tactics for different situations. In other words, students would learn how to better 
learn in the future. This belief is still prevalent today: many educators agree that personalizing learning is a 
key priority (Marshall et al., 2009). Although giving a student control over their learning has theoretical and 
intuitive appeal, its effects seem neither powerful nor consistent in the empirical literature base. A 
meta-analysis of 18 studies by Karich, et al (2014) .found - consistent with previous research (Niemiec et al., 
1996)- near zero effects for all components of instruction (pacing, time, sequence, practice, review). Thus, 
there does not seem to be an advantage to giving the learner control over any particular instructional 
component. Yet programs that offered a comprehensive approach had larger effects than practice-based 
applications, suggesting that educators should consider more comprehensive programs that provide the 
learner with a unique experience beyond what is commonly received in their classroom. Moreover, studies 
with behavioral variables had larger effects than measures of academic achievement, which suggests that 
providing learner control within educational technology may enhance engagement, but may not increase 
student skills. 

In personalizing learning with technology, learner-control is not the only dimension to be considered; 
technology in itself can control students’ learning process to some extent (Vandewaetere et al., 2011; Karich 
et al., 2014). Therefore, we distinguish between three actors influencing student learning: learner, teacher and 
technology. 

Personalizing learning from the perspective of Jearner includes learning across five aspects of program 
design: pacing, sequencing, time allotment, choice of practice items, and choice of review items (Niemiec et 
al., 1996). Pacing indicates how quickly teachers present the content to the learner. Sequencing denotes how 
teachers order information, such as when particular objectives or tasks are presented in relation to other 
objectives or tasks. Time allotment referred to the amount of time teachers give to the learner to complete the 
content in its entirety for a particular session. Practice items indicate the type and amount of practice on a 
particular objective, whereas review items are typically presented at the end of a lesson as a check for 
understanding. As mentioned above, effects of learner control are ambiguous with small effects mostly found 
on learners’ motivation and learning behavior (Corbalan et al., 2006; Karich et al, 2014). Yet just adding 
learner control to an existing learning environment does not always lead to learner behavior regulating their 
learning (Azevedo et al., 2008); students need some guidance or practice in learner control to increase their 
regulative behavior (Authors, under review). 

Personalizing learning from the perspective of teachers means that teachers determine what learners need 
to learn based on learner characteristics and adapt the content and their way of teaching (Tomlinson et al., 
2003). Teachers analyze learner characteristics such as achievement, motivation and learning behavior in 
order to adapt their teaching to what is needed for individuals or groups of students. Although research on 
personalizing learning from the teacher perspective has a long history (e.g., Snow, 1992), studies show 
ambiguous findings with respect to effects on cognitive, affective and behavioral measures (Vandewaetere et 
al., 2011). Personalizing learning from the perspective of both teacher and learners is called shared-control 
(Corbalan et al., 2006). This model combines the two approaches to personalizing learning: 
Program-controlled instruction, in which an instructional agent (e.g., computer, teacher) makes instructional 
decisions and learner-controlled instruction, in which learners makes such decisions. 

Personalizing learning from the perspective of technology refer to the adaptive qualities of the technology 
that is used to support teaching and learning (Vandewaetere et al., 2011). Empirical findings suggest that the 
use of adaptive technology in computer-based assessments has a positive impact on students’ achievements 
in various school subjects (e.g. Faber, Luyten and Visscher, 2017), but not all findings are consistent. 
Findings about the frequency of using these adaptive environments suggest that greater usage of the system 
by both teacher and students is associated with better student performance. 
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3. METHOD 


The purpose of the current study is to provide insights into the effects of personalizing learning with 
technology in secondary education, from the teacher, learner and technology perspective. The following 
research question guided our study: 

“What are the effects of personalizing learning with technology on students’ achievement, motivation 
for school and self-regulating behavior?” 


3.1 Research Design and Participants 


In total, data were collected about 42 interventions in 34 secondary schools (with 6045 students). Seven 
schools (with one intervention each) were excluded from the final analyses as these school interventions 
could not be categorized as focused on either teacher, learner or technology. So, data included in this study 
are from 35 interventions in 27 secondary schools (with 4808 students). The research design mostly included 
a setup of an experimental condition (the intervention on personalizing learning with technology) and a 
comparison condition. The latter can be students in another, but similar school (SCHOOL), in the same 
school, but in another year group (YEAR), in the same school and year group, but in another class (CLASS), 
in the same class (STUDENT) or the same student for another school subject (WITHIN). In some cases, no 
reference condition was used (ONE-GROUP). In all cases, a pre-test post-test design has been used. For an 
overview of the research design of each intervention, see Table 1, 2, 3 and 4. The research was carried out 
following the guidelines for research ethics and integrity of Utrecht University. 


3.2 Interventions on Personalizing Learning with Technology 


Interventions in personalizing learning with technology were categorized into three clusters: 1) teacher 
perspective, 2) learner perspective and 3) technology perspective. Mobile technology (laptops, tablets, smart 
phones) was used in combination with an electronic learning environment supported by different Learning 
Management Systems. 

Personalizing learning from a teacher perspective can be clusters at two levels of differentiating between 
individual students: 1) at the school level with different streams of schooling like the Dutch system of 
secondary education, which is called external differentiation, and 2) at the class level which is called internal 
differentiation (Berben and Van Teesling, 2014). The interventions examined in this study refer to internal 
differentiation. Internal differentiation by a teacher means that teachers monitor the content, the process and 
the assessment of student learning based on ability, motivation and learning behavior of their students 
(Tomlinson et al., 2003). If the learning goal is the same for the whole group of students, internal 
differentiation is called convergent differentiation; if learning goals within a group of students are different, 
international differentiation is called divergent differentiation. Finally, convergent differentiation (having the 
same learning goal for all students) requires thoughts about group constellation during group work with 
either students with the same ability level (homogeneous groups) or students with different ability levels 
(heterogeneous groups). In sum, personalizing learning from a teacher perspective have been categorized as 
1) divergent (DIV), 2) convergent with homogeneous groups (CON-HO), or 3) convergent with 
heterogeneous groups (CON-HE). 

Personalizing learning from a learner perspective refers to the degree students can control program 
aspects such as pacing, sequencing, time allotment, practicing and reviewing (Karich et al., 2014). A low 
degree of learner control (LOW) means that learners controlled pacing, whereas other program aspects have 
been set. High degree of learner control (FULL) means that all or almost all program aspects were controlled 
by the learner. In-between position (MODERATE) on the degree of control means that students controlled at 
least pacing, combined with some other program and task aspects, mostly within the limits set by the teacher 
or program. Low learner control was frequently combined with learner control of surface task characteristics 
(e.g., various tasks at the same ability level or various forms of testing the same content); full learner control 
was frequently combined with learner control of structural task characteristics (e.g., various tasks at different 
ability levels or various sources with variety of information; Corbalan et al., 2011). 
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Personalizing learning from a technology perspective refers to students completing computer-based 
assessments with either adaptive technology (ADAPTIVE) or non-adaptive technology (NON-ADAPTIVE). 


3.3 Measures 


Data about the implementation and evaluation of the interventions in school were gathered by interviews 
with teachers and students, teacher logbooks and teacher and student questionnaires. Although not all these 
were collected in all cases, in each case evaluation data have been collected from both teachers and students. 

Student achievement was measured by student scores on regular exams of the particular school subject 
related to the interventions. 

Student motivation was measured by the Situational Motivation Scale (SiMS; Guay et al., 2000). This 
questionnaire contains 16 items that are focused on why students are currently engaged in a particular activity 
concerning a school subject that is object of study. The items are clustered into four scales: 1) intrinsic 
motivation (example item: “because I think that this activity is interesting”), 2) identified motivation 
(example item: “because I am doing it for my own good”), 3) external regulation (example item: “because I 
am supposed to do it”, and 4) A-motivation (example item: “I do this activity, but I am not sure if it is worth 
it”). Students scored each item on a 5-point Likert type scale with 1= does not apply at all and 5= does apply 
to a great extent. All scales showed satisfying reliability. 

Student self-regulation was measured by a questionnaire with 32 items developed by Vandevelde et al. 
(2013). These 32 items can be clustered into 6 aspects of self-regulation: 1) task orientation (example item: 
“Before I start my schoolwork, I read the instructions carefully’), 2) planning (example item: “Before I start 
my schoolwork, I decide what to do first and what later”, 3) Persistence (example item: “I carry on until I 
finish my schoolwork’’). 4) self-efficacy regulation (example item: “I am good at connecting new things to 
what I already know’), 5) product evaluation (example item: “After finishing my school work I go over my 
answers again”, and 6) process evaluation (example item: “I ask myself: ‘Have I done it the right way?’). 
Students scored each item on a 5-point Likert type scale with 1= does not apply at all and 5= does apply to a 
great extent. All scales showed satisfying reliability. 


3.4 Analyses 


In the designs with a comparison group, analyses of covariance have been used with the both conditions as 
factors and one the effect measures (achievement, motivation or self-regulation) as dependent variable. 
Pre-test scores on the effect measures and background information of the students such as gender, age, and 
ability level (if available) were used as covariates. In cases of a one-group-only design, t-tests have been 
performed on the pre-test and post-test scores. 


4. FINDINGS 


4.1 Personalizing Learning from a Teacher Perspective 


The findings for the cluster of interventions that focus on a teacher perspective on personalizing learning are 
summarized in Table 1.Schools differed in the way they implemented personalizing learning from a teachers 
perspective with convergent differentiation in seven schools, divergent differentiation in three schools and a 
combination of both types of differentiation in one school. In all interventions, teachers designed their own 
educational materials (sometimes together with colleagues), provided instruction and guided the learning 
process of their students. Students worked individually or in small groups on assignments with guidance of 
their teacher (all interventions with convergent differentiation and the one of school 18). 

In general, the effects of the interventions on student achievement, motivation and self-regulation are 
mixed. The effects on student achievement are positive in all cases, with divergent as well as convergent 
interventions. No achievement effects were measured of the interventions with convergent differentiation 
with homogeneous groups. The findings with respect to students’ motivation are difficult to interpret, 
although interventions with convergent differentiation with homogeneous groups seem to lead to negative 
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effects. In school 17, the degree in which student groups were guided by the teachers varied a lot. 
Interventions with solely divergent differentiation do not show an effect on student motivation. The findings 
with respect to student self-regulation only show one positive effect, in school 11. Comparing both schools in 
which all three effects have been measured (school 10, DIV, and school 11, CON-HE) the intervention in 
school 11 shows effects on all three measures. The main difference between both interventions is the way of 
personalizing learning. In school 10, poor-performing students received additional instruction and high- 
performing students additional tasks; in school 11, students choose their own pacing and sequencing of 
materials sharing the same learning goal. 


Table 1. Personalizing Learning from a Teacher Perspective 


School | Research | Notudents | Personalizing Technology Achievement Motivation Self- 
Design learning regulation 
8 CLASS 362 CON-HE iPads na. 0 0 
9a CLASS 114 CON-HE iPads + 0 n.a. 
10 CLASS 106 DIV iPads + 0 0 
11 CLASS 50 CON-HE iPads + + + 
12 YEAR 104 CON-HE iPads n.a. n.a. 0 
13 CLASS 92 DIV iPads + 0 n.a. 
14 CLASS 563 BOTH-HO iPads n.a. + na 
15 CLASS 234 CON-HO Laptops n.a. +/- n.a. 
16 YEAR 195 CON-HE Laptops na. - na. 
17 ONE- 257 CON-HO No n.a. - n.a. 
GROUP 
18a ONE- 34 DIV iPads n.a. 0) 0) 
GROUP 


Note. 0= no effect; += positive effect, -= negative effect, +/-= mixed and n.a.= not applicable (i.e not measured). For 
categories of the research design and personalized learning interventions, see Method section. 


4.2 Personalizing Learning from a Learner Perspective 


The findings for the cluster of interventions that focus on a learner perspective on personalizing learning are 
summarized in Table 2.Schools differed in the degree learners controlled aspects of the program and learning 
tasks. In two interventions (22, 23), students only controlled pacing. In four interventions (26a, 26b, 28, 29b) 
students controlled pacing and sequencing of the various assignments. In one intervention (20), pacing was 
combined with a choice of practicing items. All these interventions were categorized as low learner control. 
All interventions coded as moderate combined learner control of pacing with other aspects, without learner 
control of structural program aspects. The exception was intervention 11, in which students could choose 
between tasks with different functionalities (memorizing, analyzing, understanding). Finally, all other 
interventions were categorized as full control with learner control of most program and tasks aspects. 

With respect to the effect on achievement four significant effects were found; two in interventions 
categorized as moderate and two in other interventions that —although not categorized as such- resembles the 
moderate ones. More significant effects were found with respect to student motivation, negative, positive and 
ambiguous, and spread over all kinds of interventions. Finally, with respect to self-regulation, only one 
positive effect was found (school 22, categorized as low). 
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Table 2. Personalizing learning from a learner perspective 


School Research Nstudents | Personalizing | Technology | Achievement | Motivation Self- 
design learning regulation 
19 YEAR 170 FULL BYOD n.a. - 0 
20 STUDENT 234 LOW BYOD n.a. n.a. 0 
21 CLASS 63 FULL Laptop na. - 0 
22 WITHIN 111 LOW No n.a. +/- + 
23 WITHIN 37 LOW iPads na. 0) 0 
24a YEAR 315 MODERATE iPad 0) 0 0 
24b YEAR 241 FULL iPad + - 0 
25 WITHIN 223 FULL Laptops na. + na. 
26a CLASS 2407 LOW Laptops + 0 n.a. 
26b CLASS 2407 LOW Laptops 0 0 na. 
26c CLASS 240° | MODERATE Laptops 0 0 na. 
27 CLASS 86 MODERATE BYOD n.a. 0 0 
28 YEAR 426 LOW BYOD n.a. 0 0 
18b ONE- 34¢ MODERATE iPad n.a. + 0 
GROUP 

29a YEAR 1235 | MODERATE iPad + + +f 
29b YEAR 1233 LOW iPad 0 + +/- 
9b CLASS 106 FULL CC 0 + n.a. 
30 CLASS 173 MODERATE CC + + n.a. 
31 YEAR 157 FULL BYOD n.a. 0 0 


Note. ' the same control group with 176 students; sharing the same 240 students; ? sharing the same 123 students; * 
sharing the same 34 students with intervention 18a; BYOD = Bring your own device; CC= Computer Classroom; 0= no 
effect; += positive effect, -= negative effect, +/-= mixed and n.a.= not applicable (i.e .not measured). For categories of the 
research design and personalized learning interventions, see Method section. 


4.3 Personalizing Learning from a Technology Perspective 


The interventions with respect to personalizing learning by technology refer to the use of computer-based 
assessment as a support of students’ learning processes. In the interventions, all aspects of a curriculum 
(pacing, sequencing, time allotment, practice items and review items) were personalized, either by the 
technology or the learner, as part of regular classes in a particular topic. Students can choose how much effort 
they put into completing assignments, and technology determines the other parts - if adaptive technology has 
been used. Adaptive technology also provided students with feedback about their performance (embedded 
analytics). In intervention of school 32 and 34a, non-adaptive technology has been used. In intervention 33, 
34b and 34¢c, student receive feedback about their performance (embedded analytics) and teachers are able to 
get an overview of their students’ performance (extracted analytics). In school 33, in-between feedback for 
students included information about the correct response as well as information on what students still had to 
learn. In intervention 34 and 36a, only information after completion of the computer-based assessments was 
available. 

No effects on student achievements were measured as none of the school intended to use these 
computer-based assessment to improve student achievement (see Table 3). With respect to student 
motivation, no effects or negative effects were found. These negative effects coincide with a low satisfaction 
of the students with the intervention: for both interventions, students were more satisfied with the regular 
classes on that topic; they did not see the additional value of practicing with computer-based assessments. 
With respect to self-regulation, only two significant effects are found: one positive and one negative in 
school 34. In intervention 34b, teachers use extracted analyses for their classroom instruction with various 
forms of feedback and instruction (plenary, small-group and individual settings); in intervention 34c. students 
have the possibility to ask teachers for feedback - which they generally did not do. 
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Table 3. Personalizing Learning from a Technology Perspective 


School Research Nstudents | Personalizing | Technology Achievement Motivation Self- 
design learning regulation 
32 YEAR 133 NON- iPad na. 0 0 
ADAPTIVE 
33 STUDENTt 83 ADAPTIVE Mixed n.a. - 0 
34a YEAR 77 NON- Mixed n.a. - 0 
ADAPTIVE 
34b ONE- 46 ADAPTIVE Laptop na. 0 + 
GROUP 
34c ONE- 52 ADAPTIVE Laptop na. 0 - 
GROUP 


Note. Mixed= students complete assignments on paper and on their laptop or tablet; O= no effect; += positive effect, -= 
negative effect, +/-= mixed and n.a.= not applicable (i.e .not measured). For categories of the research design and 
personalized learning interventions, see Method section. 


5. CONCLUSION 


The evaluation of personalizing learning interventions in 27 schools from three perspectives (teacher, learner 
and technology) leads to a number of conclusions and practical implications. Generally, three types of 
personalizing learning interventions seem to increase student achievement: 1) a comprehensive approach 
across the school organization and programs; 2) personalizing learning with teachers differentiating either 
convergently or divergently, and 3) learner-control interventions in which students have control of surface 
aspects such as pacing, sequencing and practicing within limits set by teachers or program. Too much 
emphasis on learner control instead of teacher control does not seem to benefit cognitive outcomes. The 
non-linear relationship between the degree of learner control and cognitive outcomes might also be an 
explanation for empirical evidence in other studies (Karich et al., 2014). 

The conclusions with respect to student motivation are less clear-cut. Similar interventions, either 
teacher- or learner-controlled, led to different effects. One explanation could be the relationship between 
learners’ autonomy, competence and relatedness influencing student motivation as indicated in the 
self-determination theory (Ryan and Deci, 2000). Providing students with control of program and task 
aspects should probably combined with differentiating between students who differ in competence. 

With respect to students’ self-regulation, only a few effects were found. We provide two explanations for 
this lack of effects. First, the interventions examined in this study might be too much focused on either 
teacher control or learner-control; a more balanced approach of personalizing learning might be more 
effective for students’ self-regulative skills. Secondly, the interventions of this study were not focused on 
increasing students’ self-regulation in particular. A more comprehensive approach in which a broach range of 
metacognitive skills of students are addressed, might lead to more positive effects (Authors., under review). 

A final critical reflection refers to the variety of the interventions examined in the current study. Even 
interventions that are similar in the way learner or teacher control was implemented vary on all kinds of 
program and task details, the length of an intervention, and the range of implementation. More research on 
underlying mechanisms of the interventions is necessary to provide a deeper understanding of possible 
benefits of personalizing learning in secondary education. 
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